{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o0VldHlp4Fs2"
      },
      "source": [
        "##### Copyright 2024 Google LLC."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "JmfzPvIU3tZy"
      },
      "outputs": [],
      "source": [
        "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AuQhzPCA4I-f"
      },
      "source": [
        "# Deploy Gemma in Vertex AI\n",
        "\n",
        "This cookbook illustrates how you can deploy the Gemma model in [Vertex AI](https://cloud.google.com/vertex-ai).\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/google-gemini/gemma-cookbook/blob/main/Gemma/[Gemma_2]Deploy_in_Vertex_AI.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rlrwJz0y4j_c"
      },
      "source": [
        "## Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "- Vertex AI\n",
        "- Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Storage pricing](https://cloud.google.com/storage/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage.\n",
        "\n",
        "## Setup\n",
        "\n",
        "Find the [Gemma 2 model card](https://pantheon.corp.google.com/vertex-ai/publishers/google/model-garden/gemma2) from the Model Garden and click \"DEPLOY\" button."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "225UGPf-2VW_"
      },
      "source": [
        "![image.png]()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KzGDQwDU56mU"
      },
      "source": [
        "Choose the model variation and the location to deploy.\n",
        "\n",
        "![image.png]()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9gNixXHd6YZm"
      },
      "source": [
        "Once the deployment is completed, you can find your endpoint [here](https://pantheon.corp.google.com/vertex-ai/online-prediction).\n",
        "\n",
        "To obtain a script for acquiring predictions from your active endpoint, navigate to the \"Sample Request\" and select \"Python\".\n",
        "\n",
        "![image.png]()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8SfMjri29YFv"
      },
      "source": [
        "Authenticate using the Google Cloud SDK"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OyUmmJEO9bSf"
      },
      "outputs": [],
      "source": [
        "!gcloud auth application-default login"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "URea-U9k7opS"
      },
      "source": [
        "Below is the sample code on Github:\\\n",
        "https://github.com/googleapis/python-aiplatform/blob/main/samples/snippets/prediction_service/predict_custom_trained_model_sample.py"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VUyuLaN97kdF"
      },
      "outputs": [],
      "source": [
        "# Copyright 2020 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License.\n",
        "\n",
        "# [START aiplatform_predict_custom_trained_model_sample]\n",
        "from typing import Dict, List, Union\n",
        "\n",
        "from google.cloud import aiplatform\n",
        "from google.protobuf import json_format\n",
        "from google.protobuf.struct_pb2 import Value\n",
        "\n",
        "\n",
        "def predict_custom_trained_model_sample(\n",
        "    project: str,\n",
        "    endpoint_id: str,\n",
        "    instances: Union[Dict, List[Dict]],\n",
        "    location: str = \"us-central1\",\n",
        "    api_endpoint: str = \"us-central1-aiplatform.googleapis.com\",\n",
        "):\n",
        "    \"\"\"\n",
        "    `instances` can be either single instance of type dict or a list\n",
        "    of instances.\n",
        "    \"\"\"\n",
        "    # The AI Platform services require regional API endpoints.\n",
        "    client_options = {\"api_endpoint\": api_endpoint}\n",
        "    # Initialize client that will be used to create and send requests.\n",
        "    # This client only needs to be created once, and can be reused for multiple requests.\n",
        "    client = aiplatform.gapic.PredictionServiceClient(client_options=client_options)\n",
        "    # The format of each instance should conform to the deployed model's prediction input schema.\n",
        "    instances = instances if isinstance(instances, list) else [instances]\n",
        "    instances = [\n",
        "        json_format.ParseDict(instance_dict, Value()) for instance_dict in instances\n",
        "    ]\n",
        "    parameters_dict = {}\n",
        "    parameters = json_format.ParseDict(parameters_dict, Value())\n",
        "    endpoint = client.endpoint_path(\n",
        "        project=project, location=location, endpoint=endpoint_id\n",
        "    )\n",
        "    response = client.predict(\n",
        "        endpoint=endpoint, instances=instances, parameters=parameters\n",
        "    )\n",
        "    print(\"response\")\n",
        "    print(\" deployed_model_id:\", response.deployed_model_id)\n",
        "    # The predictions are a google.protobuf.Value representation of the model's predictions.\n",
        "    prediction = response.predictions\n",
        "    return prediction\n",
        "\n",
        "\n",
        "# [END aiplatform_predict_custom_trained_model_sample]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mjSKdAoO76IS"
      },
      "source": [
        "Execute your request in Python"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_6J5zTyI797e"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/google/auth/_default.py:76: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK without a quota project. You might receive a \"quota exceeded\" or \"API not enabled\" error. See the following page for troubleshooting: https://cloud.google.com/docs/authentication/adc-troubleshooting/user-creds. \n",
            "  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "response\n",
            " deployed_model_id: 7748004453751455744\n",
            "--------------------------------------------------------------------------------\n",
            "Prediction Result\n",
            "--------------------------------------------------------------------------------\n",
            "  Suddenly, a flash of lightning illuminated the street, revealing a figure standing before my window.  It was a man dressed in a long coat, his face obscured by a wide-brimmed hat.  He held a lantern, the flames dancing in the wind, casting unnatural shadows around him. \n",
            "\n",
            "My heart pounded in my chest.  I knew this was no ordinary visitor.  Something about his presence sent shivers down my spine.  I couldn't help but stare, mesmerized by his strange aura.  He didn't seem aggressive, but there was an undeniable power about him.\n",
            "\n",
            "Then, he spoke.  His voice, low and raspy, seemed to echo from the depths of the storm.  \"I'm looking for a book,\" he said, his words barely audible above the roaring wind.  \"A book of forgotten magic.\"\n",
            "\n",
            "He handed me a crumpled piece of paper, the words scrawled in a language I couldn't understand.  He looked at me, his eyes piercing through the darkness.  \"I need your help,\" he said.  \"A darkness is rising, and only the power of the forgotten magic can stop it.\"\n",
            "\n",
            "My mind raced, trying to make sense of the situation.  I had never encountered anything like this before.  But something deep inside me, a sense of urgency and responsibility, compelled me to help.  I accepted the paper and promised to do everything I could. \n",
            "\n",
            "\n",
            "**What do you think happens next?** \n",
            "\n"
          ]
        }
      ],
      "source": [
        "YOUR_PROJECT_ID = \"<YOUR_PROJECT_ID>\"\n",
        "YOUR_ENDPOINT_ID = \"<YOUR_ENDPOINT_ID>\"\n",
        "YOUR_LOCATION = \"<YOUR_LOCATION>\"\n",
        "\n",
        "prompt = \"It was a dark and stormy night.\"\n",
        "\n",
        "response = predict_custom_trained_model_sample(\n",
        "    project=YOUR_PROJECT_ID,\n",
        "    endpoint_id=YOUR_ENDPOINT_ID,\n",
        "    location=YOUR_LOCATION,\n",
        "    api_endpoint=f\"{YOUR_LOCATION}-aiplatform.googleapis.com\",\n",
        "    instances={ \"inputs\": prompt, \"max_tokens\": 2048 }\n",
        ")\n",
        "\n",
        "print(\"-\"*80)\n",
        "print(\"Prediction Result\")\n",
        "print(\"-\"*80)\n",
        "print(response[0])"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "[Gemma_2]Deploy_in_Vertex_AI.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
